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PREFACE 


Conflict Resolution Advisory (CRA) software is intended to provide the air traffic 
control (ATC) specialist at air route traffic control centers (ARTCCs) with a suggested 
maneuver to prevent separation violations between aircraft. CRA calculates, 
validates, and displays a single resolution for potential separation violations predicted 
by the conflict alert (CA) function. For this purpose, it is critical to know the time 
required for the ATC specialist to process and transmit the maneuver and for the pilot 
to acknowledge it. The controller- response time, that is, the time required for a 
controller to notice, read, comprehend and evaluate the advisory, was determined in 
a previous simulation study using the prototype CRA software (DOT/FAA/NA-92/1). 
The transmission and pilot-response time was determined by analysis of actual voice 
tapes of ATC communications (DOT/FAA/RD-91/20). The purpose of this simulation 
was to validate the controller-response time found with the CRA prototype software 
using the enhanced CRA software (CRAE). Data were also collected on controller 
opinion on the CRA message format and on CRA in general. 
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EXECUTIVE SUMMARY 


Conflict Resolution Advisory (CRA) is an automated software aid for air traffic control 
specialists at air route traffic control centers (ARTCCs). CRA calculates, validates, 
and displays to the en route controller a single resolution for predicted separation 
violations detected by the conflict alert (CA) function. This simulation study was 
conducted to determine controller-response time to a CRA message. The response 
time is the total time required for controllers to notice that the advisory is present, to 
read and comprehend the text message, and to decide that the resolution is 
acceptable. 

Eight full performance level air traffic controllers were presented with six different 
two-hour traffic scenarios ranging from moderately high to very high levels of traffic 
load and complexity. They were asked to read and evaluate the CRA messages as 
they appeared. Response time, defined as the lag between the onset of the CRA 
message and the beginning of the controller's speech indicating that the resolution 
was acceptable, was measured. This response time varied from 3 to 71 seconds with 
a mean response time of 17.7 (S.D. = 8.6). While the response time was relatively 
high, the error rate was only two percent. 

Controller comments on the CRA function and message format were assessed with 
questionnaires and informal interviews. Generally, controllers were not satisfied with 
CRA. In its present form, they perceived it as unreliable and said that they would be 
reluctant to use it in actual operations. Five out of the eight controllers thought that 
the CRA message format was easy to understand. There was no consensus among 
the others as to how to improve it. 

To calculate the delay that is to be expected between CRA onset and pilot response, 
the results of this study must be applied to the results of a previous study 
(DOT/FAA/RD-91/20) that examined the time required for message transmission and 
pilot response. The analysis of the combined data from these two studies suggests 
that 49 seconds should be considered as the upper limit on the time expected to 
elapse between CRA onset and the pilot's initial input into the aircraft's controls. 
However, such a long delay may be indicated only in situations similar to the ones 
tested in this study, such as extremely high workload and high incidence of CAs for 
which CRA was not able to offer a resolution. A previous study with the prototype 
CRA software (DOT/FAA/NA-92/1), with a slightly lower workload, and a higher 
proportion of CAs for which a resolution was presented, yielded a mean response time 
of 13 seconds (S.D. = 6.2) and a more realistic error rate; these data suggested an 
upper limit of 40 seconds for the delay parameter. Presumably, controllers in the 
present study assigned a lower priority to reading the CRA message than controllers 
in the first study and this, along with the lower error rate, resulted in higher response 
times. 
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1. INTRODUCTION 


Conflict Resolution Advisory (CRA) software is intended to provide the air traffic 
control (ATC) specialist at air route traffic control centers (ARTCCs) with a suggested 
maneuver to prevent separation violations between aircraft. CRA calculates, 
validates, and displays a single resolution for potential separation violations predicted 
by the conflict alert (CA) function. 

As with any time-critical warning system, the algorithm must take into account the 
time required for the operator to use the system. The lag between the time that the 
CRA message appears on the controller's display and the time that the aircraft begins 
to maneuver will affect the number and type of potential resolutions. It is therefore 
critical to know the controller-response time, the transmission time and the pilot- 
response time. The transmission and pilot-response times, that is, the time from the 
beginning of the controller's message to the end of the pilot's correct 
acknowledgment, was determined by analysis of voice tapes of ATC communications 
(DOT/FAA/RD-91/20). Results of informal interviews and a small pilot study support 
the assumption that further delays between acknowledgment and aircraft maneuver 
are negligible. 

This simulation examined the controller-response time, that is, the total time required 
for controllers to notice that the advisory is present, to read and comprehend the text 
message, and to decide that the resolution is acceptable. This decision process is 
required because the CRA software does not take into account all of the information 
available to the controller. The controller needs to ascertain that a resolution is valid, 
strategically acceptable, and does not turn the aircraft into restricted airspace, a 
thunderstorm cell or other hazard. 

The controller-response time was determined in a previous simulation study using the 
prototype CRA software (DOT/FAA/NA-92/1). The purpose of this simulation was to 
validate the controller-response time found with the prototype software using the 
enhanced software (CRAE). Controllers were also asked to express their opinions on 
the CRA message format and on CRA in general. 
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2 . SIMULATION STUDY 


2.1 Method 

This simulation study was conducted over a three-week period at the DYSIM 
laboratory facilities of the FAA Technical Center in Atlantic City, NJ. 

CRA Display. 

The CRA messages appeared on the CA tabula" list. Figure 2-1 shows examples of 
CRA messages as they appeared to the test controllers. A single resolution was 
presented for each CA. For single-pair conflicts (Examples 1 and 2), the resolution 
was displayed directly below the CA list containing the aircraft identifications for the 
two aircraft in conflict. For multiple-pair conflicts (Example 3), the list of conflict pairs 
was first followed by a line consisting solely of the letter "M" for multiple conflict, and 
then by the recommended resolution. 

For the resolution advisory, each aircraft involved In the conflict was displayed on one 
line, followed by the recommended maneuver. A maneuver consisted either of a 
horizontal maneuver, a vertical maneuver, or a "maintain" maneuver. A horizontal 
maneuver consisted of a right ("R") or a (eft ("L") turn followed by the number of 
degrees (Examples 1 and 3). A vertical maneuver consisted of a "descend" or "climb" 
expressed with the appropriate arrow (t or i) followed by the altitude in hundreds of 
feet (Example 2). If the aircraft was already climbing or descending, the vertical 
maneuver issued would be a "maintain". This was displayed as an "M" followed by 
the altitude in hundreds of feet (Example 2). 

Sometimes, only one of the aircraft involved in the conflict was required to maneuver 
to resolve the conflict (Example 1). Other times, more than one aircraft involved in 
the conflict was required to maneuver. This was indicated by the letter "J" (for joint 
maneuver) preceding each resolution line (Examples 2 and 3). 

For some conflicts, CRA was not able to provide a resolution. This was indicated by 
the message "NO-RES" below the CA tabular list, followed by the reason. "NO-RES 
AVAIL" was displayed when no resolution was available that passed the validation 
procedure of the CRA logic or when the conflict involved an uncontrolled aircraft 
below 18,000 ft with a nondiscrete beacon code or no flight plan (Example 4). "NO- 
RES IF" (for interfacility) was displayed when the conflict involved an aircraft that was 
under the control of another facility. "NO-RES SEP" was displayed when a separation 
violation had already occurred or when a resolution could not be computed before a 
separation violation would have occurred. 
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EXAMPLE 1: SINGLE PAIR CA. SINGLE HORIZONTAL CRA 


CONFLICT ALERT 
.AAL210 UAL202 01 04’ 

AAL210 
UAL2022 R30^ 


EXAMPLE 2: SINGLE PAIR CA, JOINT VERTICAL CRA 

CONFLICT ALERT 
.AAL210 UAL202 
J AAL210 M270* 

J UAL202 t290 


EXAMPLES: MULTIPLE-PAIR CA, JOINT HORIZONTAL CRA 

CONFLICT ALERT 
.AAL210 UAL202 
.UAL202 DAL301 
M® 

J AAL210 
J UAL202 R30 
J DAL301 L30 


EXAMPLE 4: NO RESOLUTION A VAILABLE 

CONFLICT ALERT 
.AAL210 UAL202 
NO-RES AVAIL 

FIGURE 2-1. EXAMPLES OF CRA MESSAGES 


’ CA tabular list with intersector notation, 

^ AID for CRA aircraft. 

^ Position for display of horizontal maneuver - right or left turn in degrees. 
* Vertical maneuver - climb, descend, or maintain. 

® Designates multiple-pair conflict. 
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As a result of the previous simulation study using the CRA-prototype software 
(DOT/FAA/NA-92/1), the maximum display time for a resolution was 24 seconds. 
When a conflict persisted beyond 24 seconds, the resolution lines were replaced by 
the message "NO-RES TOUT" directly below the CA list indicating that the time for 
issuing the resolution had run out. This time-out message persisted until the conflict 
was resolved (but see Section 2.2). 

If the conflict configuration changed after the first 12 seconds of displaying a 
resolution, any new or changed data in the CRA tabular list blinked and the list 
including the new data was displayed for 24 seconds. If the newly calculated CRA 
was inconsistent with the previous one, i.e., it required a change in type or dire tion 
of maneuver, or if no valid resolution could be determined, the message "NO-RES 
AVAIL" was displayed. 

1^ there was more than one conflict configuration at any one point, the CA and CRA 
list of a configuration immediately followed the CA and CRA list of a previous 
configuration. 

Subjects. 

Eight full performance level (FPL) controllers were selected from a list of volunteers 
from the Ft. Worth ARTCC. The controllers were certified and current on the sectors 
that they worked during the test. 

Airspace. 

The simulated airspace consisted of three contiguous test sectors. Two of these 
sectors were high altitude sectors (Dallas High and Ardmore High) and the third was 
a low altitude sector (Frisco Low). These sectors were supported by a controller on 
a "ghost" sector; the ghost sector initiated and received aircraft from the three test 
sectors. 

Scenarios. 

Six different scenarios, each approximately 1 1/2 to 2 hours in duration, were 
developed based on the actual traffic from the sectors chosen for the simulation. To 
ensure the occurrence of conflicts, the scenarios were constructed to range from 
moderately high to very high workload (in terms of number of aircraft and traffic 
complexity). The scenarios contained emergencies and other unusual conditions (e.g., 
aircraft with no radio) and one of the scenarios contained significant weather 
conditions. Simulated aircraft were operated by experienced DYSIM pilots. The 
scenario developers worked with the DYSIM pilots and were able to make immediate 
adjustments to the traffic loads, as necessary. 

Data Collection and Procedure. 

The simulation took place over the course of three weeks. During each week, the six 
scenarios were run over three days. On each day, two different scenarios were run 
between 6:00 p.m. and midnight. Data from approximately 68 hours of simulation 
were collected. 
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Controllers were instructed to work as they normally would in that sector, following 
standard operating procedures. Each controller worked alone as there were no D-side 
(radar associate) controllers. Before the first test session, the controllers were briefed 
on the functions and limitations of the CRA function and were given a description of 
the display. 

For each CA, the controllers were asked to key the microphone and state whether it 
was a nuisance alert (when separation was already assured or the conflict pair was 
not in their airspace), an alert involving VFR aircraft (i.e., an uncontrolled aircraft with 
no discrete beacon code or no flight plan filed flying under Visual Flight Rules), or a 
true CA. For a true CA, they were asked to read the callsigns of the conflicting 
aircraft and the CRA message. If a CRA maneuver was present, the controllers were 
asked to evaluate it, that is, to key the microphone and state whether or not the 
resolution was acceptable. They were explicitly told that they did not have to use a 
resolution even if they judged it to be acceptable. 

For the analysis of the controller-response time, controllers' comments (including all 
communications with the D^SIM pilots and other controllers), the CA/CRA tabular list 
as it appeared to the controllers on the Plan View Display (PVD), and the PVD time 
were recorded on videotape. 

To assess controllers' subjective impressions regarding the realism and the complexity 
of the traffic as well as the workload involved in controlling it, controllers filled out 
questionnaires after each test session. Controllers' overall opinions regarding the 
realism of the equipment and the communications were collected with a post¬ 
simulation questionnaire. The same questionnaire also asked controllers' opinions 
regarding the CRA message format and CRA in general. 

2.2 Results 

CRA Display. 

Resolution Maneuvers . There were 1,411 instances of CA during the approximately 
68 hours of simulation. Table 2-1 shows the distribution of resolutions by type of 
maneuver. 

Forty-three percent (604) of the advisories contained a maneuver. Over 85% (521) 
of these were single maneuvers, and fewer than 15% (83) were joint maneuvers 
requiring more than one aircraft to take action. Of the single maneuvers, close to half 
(237) consisted of a "maintain", the rest were almost equally distributed among 
vertical and horizontal maneuvers. Of the 150 vertical maneuvers, most (147) were 
descents, only three consisted of a climb. Of the 134 horizontal maneuvers, 82 were 
left turns and 52 were right turns. 

For 53% (746) of the CAs, no resolution could be advised. In the vast majority (545) 
of these cases no resolution was available because a VFR aircraft (275 cases) was 
involved or because the CRA function was not able to calculate a valid resolution. 
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In four percent of the CAs, a malfunction occurred resulting either in the aircraft 
identifications (AIDs) appearing alone or in "NO-RES TOUT" appearing as the initial 
CRA message. 


TABLE 2-1. RESOLUTIONS BY TYPE OF MANEUVER 


Maneuver Tvoe 


Number 

Percentaae of Total 

Single Maneuvers 


521 

37 

Maintain 

237 


17 

Descent 

147 


10 

Climb 

3 


.2 

Left Turn 

82 


6 

Right Turn 

52 


4 

Joint Maneuvers 


83 

6 

No Resolution Advisory 


746 

53 

NO-RES AVAIL 

545 


39 

NO-RES IF 

45 


3 

NO-RES SEP 

156 


11 

Display Malfunction 


61 

4 

NO-RES TOUT 

33 


2 

AIDS only 

28 


2 


Display Clutter . One of the concerns about the CRA display was that it may add too 
many lines to the tabular list and be too cluttered to read when there was more than 
one conflict. In order to examine this issue, the number of lines of text that appeared 
on the CRA display for each new conflict were counted. For one conflict involving 
two aircraft, three lines of text (including the CA list) would appear if a resolution was 
displayed, and two lines of text (CA list and "NO-RES") would appear if a resolution 
was not presented (see Figure 2-1). As can be seen in Figure 2-2, the tabular list 
was blank when 64% of the CRA messages appeared (as indicated by 900 instances 
of an initial display of fewer than four lines). In 511 (36%) of the conflicts, however, 
there were already four or more lines of text on the tabular list when the CRA 
message indicating a new conflict appeared. 
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FIGURE 2-2. NUMBER OF LINES OF TEXT IN INITIAL CRA DISPLA Y 


All CRA messages disappeared upon conflict resolution (as CA disappeared) or, after 
24 seconds, changed to "NO-RES TOUT". "NO-RES TOUT", which was supposed to 
disappear after conflict resolution, persisted for a minimum of 10 seconds and a 
maximum of 43 minutes. On average, it disappeared after 68 seconds (S.D. = 147 
seconds). The long maximum is assumed to be due to a display malfunction. In nine 
percent of the CAs, a malfunction caused the time-out message to disappear, and 
reappear several seconds later. 

Controller Response. 

Table 2-2 categorizes the eight controllers' responses to the 1,411 alerts. CRA 
messages generated as a result of a nuisance CA or when all of the conflict aircraft 
were not in voice communication with the test controller (i.e., under track control 
only) were not evaluated by the controllers. An additional 479 CAs/CRAs were either 
not noticed or not commented on by controllers. In 396 instances of CA, the 
controller said that CRA did not offer a resolution (i.e., he or she read "No Res" into 
the tape). There were 258 instances of CRA where the resolution was evaluated by 
controllers as either acceptable or unacceptable. 
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TABLE 2-2. CONTROLLER RESPONSES TO CA/CRA 


Controller Response Frequency 


Resolution Acceptable 193 

Nuisance Alert 76 

No Comment on CA/CRA 479 

Aircraft Under Track Control Only 130 

No Resolution Available 396 

Resolution Unacceptable 65 

Other 72 

Total 1.411 


Acceptable Resolutions . In 14% (193) of the total CAs (75% of the 258 CRA 
messages that were evaluated by the controllers), the resolution was considered 
acceptable. Twenty-one of these resolutions were not explicitly evaluated by the test 
controllers but were considered acceptable because the controller issued the same 
maneuver just prior to the CRA display onset, or immediately after the CRA message 
appeared (but without any indication that the controller read the resolution). In 16 
instances, a malfunction caused a delay in the CRA display so that the resolution 
appeared seconds after the onset of the CA. The remaining 156 instances in which 
the resolutions were explicitly evaluated and the CRA display worked properly were 
analyzed for controller-response time. 

The time between the onset of the CRA message and the onset of controller speech 
indicating an acceptable resolution was measured. For example, if the controller's 
response was, "CRA wants to turn United 123 right thirty degrees . . . yeah, that will 
work," the end of the response time was measured as the beginning of the "yeah." 
This response time varied from 3 to 71 seconds (see Table 2-3) with an average of 
18 seconds. Table 2-4 shows the response times at several percentiles. 

TABLE 2-3. CONTROLLER-RESPONSE TIME (IN SECONDS) 


Minimum 3.00 

Maximum 71.00 

Mean 17.74 

Standard Deviation 8.65 

Total Number of Observations =156 
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TABLE 2-4. CONTROLLER RESPONSE- TIME (PERCENTILES) 


5th 

9.0 

10th 

10.0 

50th 

16.0 

90th 

27.0 

95th 

30.0 


The variability among controllers in terms of response times was small. The mean 
response time for individual controllers ranged from 14.6 to 20.7 seconds. 

Nuisance Alerts . Only five percent of the CA instances were classified by the 
controllers as nuisance alerts. The resolutions to these nuisance CAs were not 
evaluated by the controllers. 

No Comment . In 479 (34%) of the 1,411 instances of CA the controller did not 
comment on the CA or CRA message. Most (371) of these instances were due to the 
controller not mentioning that a CA was occurring. The remaining of these instances 
(108) were due to the controller noting that there was a CRA message but not saying 
whether or not the resolution was acceptable. An example of such a controller's 
response is "CRA says to turn American 123 right 30 degrees, but I'd rather climb 
him ...". 

Advisories for Aircraft Under Track Control Only . Nine percent of the CAs involved 
aircraft that were not in voice communication with the test controller. Resolutions 
that were presented to the test controller, but were intended for aircraft under track 
control only (and not in voice communication with the test controller) were not 
evaluated by the controllers. 

No Resolution Available . For 396 (28%) of the CAs, the controllers stated that CRA 
did not generate a resolution. The majority (275) of these cases involved VFR aircraft. 

Unacceptable Resolutions . In four percent (65) of the total CAs (25% of the 
evaluated CRAs), the resolution was considered unacceptable. This does not include 
the instances in which the CRA resolution given to the controller involved maneuvers 
for aircraft not in voice communication with the test controller. The controllers' 
reasons for judging the resolutions to be unacceptable varied. In most cases, the 
controllers thought that the stated resolutions would either not be of sufficient 
magnitude (as in the case of a turn) to resolve the conflict, or would cause a conflict 
with a neighboring aircraft. Very few of the resolutions were judged to be 
unacceptable for reasons other than traffic avoidance (e.g., due to weather or 
restricted airspace). However, since the controllers were not required to state the 
reasons why each resolution was unacceptable, a meaningful statistical breakdown 
of these reasons is not possible. 
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Other. In five percent of the total CAs, the CRA message disappeared or changed to 
"NO-RES TOUT" (time out) before the controller could read it. Fewer than one 
percent could not be read because the tabular list was too cluttered. 

Controller-Response Errors. 

There were only five instances (two percent of the 258 CRAs that were evaluated) 
in which the controller read information that was different than what appeared on the 
tabular list. In three cases, the controller misread (or misspoke) the aircraft call sign 
and in one case, the controller reported the wrong maneuver. In the fifth case, the 
maneuver was assigned to the wrong aircraft; for example, in a conflict between UAL 
123 and AAL 456, the controller read "Right 40 for AAL 456" when the resolution 
was "Right 40 for UAL 123". 

This two-percent error rate is substantially lower than what would be expected for 
such a task; it is also lower than the 24 errors (16% of the 125 CRAs that were 
evaluated, excluding four errors where no resolution was reported when one appeared) 
observed in the previous study (DOT/FAA/NA-92/1). There are two probable reasons 
for this difference. First, changes in the display made some of the errors less likely. 
For example, in the previous display an "R" meant both that the resolution involved 
only one maneuver (as opposed to a joint maneuver) and a right turn, depending on 
the location of the "R". This ambiguity was removed from the display of the 
enhanced software. Second, controller-response time was substantially longer for 
the second test than for the first test. Despite other possible reasons for this longer 
response time, fewer errors would be expected because of the well-established 
speed/accuracy trade-off in such tasks. 

Post-Run Questionnaires. 

Controllers rated the realism of the traffic situations after each session on a scale of 
1 to 5. A score of 1 was equivalent to "very unrealistic", 3 was "moderately 
realistic" and 5 was "very realistic". The arithmetic mean (average) of the 48 scores 
collapsed across scenarios and sectors was 3.9 with a standard deviation (S.D.) of 
.98. Scores ranged from 1 (one single response) to 5. 

Workload was measured by having controllers rate the volume and the complexity of 
traffic. Traffic estimates ranged from 70% to 150% of the traffic volume normally 
seen in the sectors simulated, with a mean of 111% (S.D. = 19.52). Traffic 
complexity ratings ranged from 1 (very low complexity) to 5 (very high complexity), 
with a mean of 4.1 (S.D. = .74). 
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3. POST- TEST QUESTIONNAIRE RESUL TS 


After the simulation, controllers were given a questionnaire asking them to rate the 
overall realism of the communications, the communications equipment, and the 
displays and controls. Furthermore, they were given the opportunity to make 
suggestions regarding the format and content of the CRA messages and to express 
their opinions on CRA in general. 


3.1 Controller Comments on Realism of Simulation 

With the anchors "very unrealistic", "moderately realistic", and "very realistic" for the 
ratings of 1, 3, and 5, respectively, the eight controllers gave both the 
communications and the communications equipment a mean rating of 3.75, each with 
a standard deviation (S.D.) of .83. The displays and controls were given a realism 
rating of 4.25 (S.D. = .66). 

3.2 Controller Comments on CRA 

Comments on CRA Message Format and Content. Five of the eight controllers 
thought that the format of the CRA messages was easily understood. Three 
controllers thought that it was hard to understand. 

Only one of the controllers remembered having observed a blinking update to a CRA 
message. Six of the controllers approved of the blinking update presentation in 
principle, with R- and D-side capability to suppress blinking. Two controllers did not 
express an opinion. 

All controllers (with the exception of one who didn't answer the question) approved 
of the fact that "NO-RES TOUT" was presented steady state rather than blinking. 
Two controllers thought that all CRA messages should change to "NO-RES TOUT" 
after 24 seconds, just as they observed in the test. Three controllers, however, 
would have liked the message to disappear instead, one of them suggesting that the 
line remain blank. One controller, for the "NO-RES" messages, would have liked to 
be able to read why no resolution was available until the end of the CA. Another 
would have preferred to also keep the advisories, marked "TOUT". A third suggested 
that the advisory be reevaluated and a newly calculated resolution (new or old) be 
displayed again for 24 seconds. 

Two controllers indicated that the "NO-RES SEP" message should appear only when 
separation between aircraft had already been broken. Two others thought the 
message should only appear when the CRA function was not able to calculate a 
resolution that ensured separation. One controller would have liked the "NO-RES 
SEP" message to appear in both cases. Two controllers suggested a blank line below 
the CA in either case. One controller did not express a preference. 
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For conflicts involving an uncontrolled aircraft with no discrete beacon code or no 
flight plan, none of the controllers liked the "NO-RES AVAIL" message displayed 
during the simulation. Instead, four controllers would have preferred no message at 
all, three controllers suggested a blank space appear under the CA list, and one 
controller would have preferred the message "NO*RES VFR". 

Overall, there was concern that the CRA lists were too long and cluttered and that 24 
seconds was not enough time to evaluate a resolution, especially when there were 
several conflicts present at the same time. 

Comments on CRA in General. To the question "How will CRA be used by 
controllers?", six controllers replied that they would not use it as presented because 
it was not sufficiently reliable. One controller mentioned that he would never use it, 
because he would not want to take his eyes off the aircraft in conflict to read the CRA 
list during a CA. Only one controller mentioned that he would use CRA occasionally, 
but only as a back-up aid. One controller added that even if CRA supplied reliable 
resolutions, it would be useful only after extensive controller training in reading and 
evaluating CRA maneuvers. 

Controllers also mentioned that CRA may be counter-productive for two reasons. 
First, evaluating the CRA message in a moment of crisis consumes valuable mental 
resources and time that would be better spent on thinking of a resolution. Second, 
discrepancies between the displayed resolution and the one controllers thought of 
independently might undermine controllers' confidence in their own resolutions. 

One last concern mentioned by the controllers was that continuous use of CRA may 
weaken controllers' problem-solving and planning skills. If controllers learn to rely on 
automated systems providing them with resolutions in critical situations, what will 
they do if the system fails? 


i 


14 






4. SUMMARY AND CONCLUSIONS 


4.1 Calculation of the Delay Parameter 

The purpose of this study was to provide data for the computation of the delay 
parameter to be used in the CRA algorithm. This delay parameter would provide the 
expected time lapse between the onset of the CRA message and the pilot's input into 
the aircraft's controls. An estimate of this delay must take into account several 
factors. These factors include the controller's response time, the time required for 
successful transmission of a controller's message to the pilot, and the pilot's response 
time. The results of this study showed that an average of 18 seconds is required for 
a controller to read the CRA message and decide that it is usable. 

To determine pilot-response time, a study was conducted using voice tapes from 
ARTCCs (DOT/FAA/RD-91/20, August 1991). Pilot-response time was combined with 
transmission time and measured from the beginning of the controller's transmission 
of a maneuver required for traffic avoidance to the end of the pilot's correct 
acknowledgement. The data from that study are summarized in Table 4-1 and Table 
4-2. Personal communications with experts (such as the National Resource Specialist 
in Flight Management) and an informal pilot study supported our assumption that by 
the end of the verbal acknowledgement, the pilot will initiate an input into the controls 
(whether or not the pilot communicating is also the pilot flying). 


TABLE 4-1. PILOT-RESPONSE TIME (IN SECONDS) 


Minimum 4.00 

Maximum 40.00 

Mean 10.85 

Standard Deviation 5.91 


Total Number of Observations = 80 


TABLE 4-2. PILOT-RESPONSE TIME (PERCENTILES) 


5th 

5.0 

10th 

6.0 

50th 

9.0 

90th 

17.0 

95th 

22.5 


One way of estimating the CRA delay parameter is to add the mean or 90th percentile 
pilot-response time to the mean or 90th percentile controller-response time. However, 
a more appropriate statistical analysis considers all possible pairings of these two data 
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sets and calculates the percentiles for the sum of the response times over all pairings. 
This method provides the best estimates available for the combined percentiles, 
assuming that the pilot and controller-response times are independent. Table 4-3 
presents these percentiles with the upper and lower confidence limits and the mean 
with the 90th and 95th percent confidence limits. As can be seen in the table, we 
are 95% confident that the mean of this total time is greater than 26.72 and less than 
30.46, and that the 95th percentile is less than or equal to 54 seconds. 


TABLE 4-3. COMBINED CONTROLLER/PILOT-RESPONSE TIMES (IN SECONDS) 


Percentiles 



5th 

10th 

90th 

95th 

Point Estimates (in seconds) 

16.5 

18 

41 

49 

Confidence Limits: 

Lower .05 

15.5 

17 

— 

— 

Lower .10 

16 

17 

— 

— 

Upper .10 

— 

— 

43 

53 

Upper .05 

— 

— 

45 

54 


Mean Total Response Time = 28.59 
95% Confidence Limits = (26.72, 30.46) 

90% Confidence Limits = (27.02, 30.16) 

These data suggest that, under the conditions tested, the upper limit on the time 
expected to elapse between the onset of the CRA display and the time the pilot makes 
an input into the aircraft's controls should be 49 seconds. However, there are other 
factors that need to be taken into account when assigning a value to the delay 
parameter. 

First, the findings of the simulation study of the CRA prototype software 
(DOT/FAA/NA-92/1) should weigh heavily in determining the delay parameter. On the 
basis of this earlier study, a delay parameter of 40 seconds was recommended. 
(Reasons for the difference between the results of the two studies are discussed in 
the next section.) Second, although neither study was large enough to study 
response times as a function of level of workload, it is reasonable to assume that 
relatively long response times would be most prevalent under extremely high workload 
conditions. By design, the workload conditions used in the study were much more 
representative of "worst-case" than of normal conditions. (A more realistic simulation 
of controller workload would have resulted in a simulation study that was prohibitively 
long and expensive.) Third, increasing the delay parameter would result in an even 
higher percentage of situations in which the CRA function could not offer a resolution 
to the controller. 
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4.2 Comparison of Results of the Two Simulation Studies 

The controller-response times observed in the present study were considerably longer 
than the ones observed in the simulation study of the prototype CRA software 
(DOT/FAA/NA-92/1). The previous simulation study was conducted in two parts. 
Part I (Week 1) examined how controllers use the CRA display and Part II (Week 2) 
examined the time required for a controller to read and evaluate the CRA text 
message. The results of Week 2 indicated that an average of 13 seconds (S.D. = 6.2) 
should be expected for the controller to read the CRA message and decide that it is 
usable. Ninety-five percent of the controllers' vocal responses were initiated within 
26 seconds. In the present study, the mean controller-response time was 18 
seconds, and the response time at the 95th percentile was 30 seconds. 

There are many possible reasons for the higher response times found for this 
simulation study compared to the previous study. First, although the ratio of 
acceptable to unacceptable resolutions was very similar in the two studies 
(approximately 3:1), controllers found that no resolution was available much more 
frequently in this simulation than in the first study. This was, in part, due to the 
greater number of VFR aircraft in the present simulation, and in part to the longer 
delay parameter for the controller/pilot- response time assumed by the CRA algorithm 
as a result of the previous study (40 seconds instead of 24 seconds). In Part II of the 
first simulation, controllers found no resolution available for 11 % (38) of all CAs 
(358). In the present study, controllers indicated that no resolution was available for 
28% (396) of all CAs (1,411). Such a low payoff further lessens the controllers' 
inclination to take their eyes off of the conflict to look at the tabular list. Also, 
because a blank line in the prototype software was replaced by the text line "NO-RES 
AVAIL" in the enhanced software (in instances in which no satisfactory resolution 
could be calculated), and because there was a higher number of these instances in the 
second study than in the first study, the CRA display was more cluttered in the 
second study. (See also the discussion of display clutter in section 2.2.) 

Second, it is likely that higher workload levels contributed to the higher response 
times found in the second study. Controllers estimated traffic load and traffic 
complexity only slightly higher in the second study-traffic load, estimated in percents 
of the traffic normally found in the same sector, ranged from 70% to 150% with an 
average of 111 % compared to 60% to 150% with an average of 105% in Part II of 
the first study, and traffic complexity was rated an average of 4.1 (on a scale of 1 to 
5) compared to 3.4 for the first study. However, there is some evidence that 
suggests that workload was higher in the present study than in the previous study. 
Specifically, with approximately 20 CAs per hour in the second study, as compared 
to approximately 10 CAs per hour in the first study, the number of CAs encountered 
by the controllers in the second simulation was dramatically higher. Although the 
number of CAs may have been slightly underreported in the first study because the 
total number of CAs was tallied by observers during the experiment rather than 
analyzed on videotape, the difference is still indicative of a higher level of workload 
in the second study. 
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It is likely that, given the extremely high workload situations and the fact that CRA 
generated a resolution for less than half of the CAs, evaluation of the CRA message 
was far lower on the controllers' lists of priorities than controlling traffic and 
maintaining proper separation. This conclusion is supported by the fact that the 
percentage of CRA messages in which the controller did not address the CA/CRA is 
higher in the second study (371 instances or 26% of all CA/CRAs) than it was in the 
first study (45 instances or 13% of all CA/CRAs). Additional support comes from the 
similarity of response times of the present study to the response times in Part I (Week 
1) of the previous study. In Part I of the test of the CRA prototype, use of the CRA 
display was optional. Response times were measured for the instances in which the 
controller looked at and issued the CRA resolution. Out of 300 CAs, there were only 
14 such instances, with an average of 18 seconds and a standard deviation of 10.4 
seconds. While this procedure had the advantage of realism, it had to be discontinued 
because it generated too few data points for a valid parameter study; only the second 
part of the simulation was considered for determining the response times. 
Nevertheless, the fact that the average response time in Part I of the earlier study was 
identical to the one found in the present study suggests that the controllers in this 
study, despite instructions to the contrary, used the CRA display much like the 
controllers in Part I of the previous study did-they reviewed the CRA messages as 
time permitted without an intention to use the resolutions. 

One last factor that may have contributed to the long response times in the present 
study is the unrealistically low error rate of approximately two percent. The 16% 
error rate found in the first study is much more in line with what would be expected 
for the performance of such a complex task. These data seem to show the same 
speed-accuracy trade-off found with many similar tasks. 

In conclusion, the response times found in both the present and the earlier studies 
should be considered indicative of what would be expected in conditions that are 
similar to the ones tested. An average controller-response time of 18 seconds could 
be expected in a situation similar to the one of the present study: extremely high 
workload situations, a significant proportion of situations in which no resolution is 
available (in this case caused, in part, by the relatively high proportion of VFR traffic), 
and, consequently, a relatively low priority assigned by controllers to read the text 
message. A slightly shorter response time, such as the average of 13 seconds found 
in Part II of the first study, could be expected with slightly lower workload and a 
higher proportion of CAs in which a resolution is presented. Changes in other key 
factors, such as display characteristics, perceived reliability of CRA, etc., would also 
be expected to affect controller-response time. 

As to how controllers would use an automated aid such as CRA, Part I of the 
simulation study using the prototype CRA software showed that the controllers' initial 
response to CRA was to resolve an impending conflict and then, if there was time, 
examine the CRA message. This response would be expected from controllers who 
have never used CRA. Experience with such a system would be expected to change 
the way controllers use it. If the system performs flawlessly, then controllers will 
trust it and will not hesitate to use the advisories, particularly when a CA was not 





anticipated or the controller does not already have an avoidance maneuver planned. 
This trust in the system would result in a relatively short response time. However, 
if the system generated maneuvers that the controllers find unacceptable, then the 
controllers would be reluctant to use it, and would probably only read the display out 
of curiosity or when caught off-guard by a CA. It is worth considering that, if the 
controller blindly issues a computer generated solution (whether out of inexperience, 
high workload, panic, fatigue, or a combination of these factors), the outcome of the 
situation will depend upon the reliability of the conflict resolution system. Also, the 
effects of long-term use of such a system on controller problem-solving and planning 
skills have not yet been systematically studied. 
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